ready4 – Data - management

Docs: Standardise Variable Values With Fuzzy Logic And Correspondence Tables

Thu, 30 Nov 2023 00:00:00 +0000

This below section renders a vignette article from the costly library. You can use the following links to:

view the vignette on the library website (adds useful hyperlinks to code blocks)
view the source file from that article, and;
edit its contents (requires a GitHub account).

library(ready4)
library(ready4use)
library(costly)

In brief

The steps described and explained in this vignette can also be (more succinctly) accomplished with the following code.

X <- CostlyCountries() 
X <- renew(X, type_1L_chr = "default") 
X <- renew(X, "jw", type_1L_chr = "slot", what_1L_chr = "logic") 
X <- renew(X, T, type_1L_chr = "slot", what_1L_chr = "force")
X <- ratify(X)

Create project

We begin by creating X, an instance of the CostlyCorrespondences module.

X <- CostlyCorrespondences()

Supply seed dataset

We begin by creating a CostlySeed module instance that includes a dataset containing our variable of interest (in this case, countries). The dataset needs to be paired with a dataset dictionary using the Ready4useDyad module from the ready4use R library. You can supply a custom standards dataset (a tibble), dictionary (a ready4use_dictionary) and the concept represented by our variable of interest using a command of the following format.

# Not run
# A <- CostlySeed(Ready4useDyad_r4 = Ready4useDyad(ds_tb = tibble::tibble(), dictionary_r3 = ready4use_dictionary()), include_chr = c("Country"), label_1L_chr = "Country")

The add_default_country_seed function will perform the previous step using values that pair the world.cities dataset of the maps R library with an appropriate dictionary and specifies countries as the concept we will be standardising.

A <- CostlySeed() %>% add_default_country_seed()

We can now inspect the first few records from our labelled seed dataset.

renewSlot(A, "Ready4useDyad_r4", type_1L_chr = "label") %>%
exhibitSlot("Ready4useDyad_r4", display_1L_chr = "head", scroll_box_args_ls = list(width = "100%"))

Dataset
City name	Country name	Population size	Latitude coordinate	Longitude coordinate	Is the nation's capital city
'Abasan al-Jadidah	Palestine	5629	31.31	34.34	0
'Abasan al-Kabirah	Palestine	18999	31.32	34.35	0
'Abdul Hakim	Pakistan	47788	30.55	72.11	0
'Abdullah-as-Salam	Kuwait	21817	29.36	47.98	0
'Abud	Palestine	2456	32.03	35.07	0
'Abwein	Palestine	3434	32.03	35.20	0

We can also inspect the data dictionary contained in A.

exhibitSlot(A, "Ready4useDyad_r4", type_1L_chr = "dict", scroll_box_args_ls = list(width = "100%"))

Data Dictionary
Variable	Category	Description	Class
name	City	City name	character
country.etc	Country	Country name	character
pop	Population	Population size	integer
lat	Latitude	Latitude coordinate	numeric
long	Longitude	Longitude coordinate	numeric
capital	Capital	Is the nation's capital city	integer

We now specify the dictionary category that corresponds to the variable we wish to standardise (“Country”). We need to use the same category name to label the results objects that we generate in subsequent steps.

A@include_chr <- A@label_1L_chr <- "Country"

We now add A to X.

X <- renew(X, A, what_1L_chr = "seed")

Specify standards

We next must specify a dataset that includes the complete list of allowable variable values.

This workflow for this step is similar to that for specifying standards, except that instead of a CostlySeed module we use a CostlyStandards module.

# Not run
# Y <- CostlyStandards(Ready4useDyad_r4 = Ready4useDyad(ds_tb = tibble::tibble(), dictionary_r3 = ready4use_dictionary()))

In many cases using the ISO_3166_1 dataset from the ISOcodes library will be the optimal choice for the standardised form of country names. We can use the add_country_standards function to pair this dataset with its dictionary and create B, a CostlyStandards module instance.

B <- CostlyStandards(Ready4useDyad_r4 = Ready4useDyad() %>% add_country_standards())

We can inspect the first few cases of the labelled version of the dataset in B.

renewSlot(B, "Ready4useDyad_r4", type_1L_chr = "label") %>% 
  exhibitSlot("Ready4useDyad_r4", display_1L_chr = "head", scroll_box_args_ls = list(width = "100%"))

Dataset
Alpabetical country code (two letters)	Alpabetical country code (three letters)	Numeric country code	Country name	Country name (official)	Country name (common alternative)
AW	ABW	533	Aruba	NA	NA
AF	AFG	004	Afghanistan	Islamic Republic of Afghanistan	NA
AO	AGO	024	Angola	Republic of Angola	NA
AI	AIA	660	Anguilla	NA	NA
AX	ALA	248	Åland Islands	NA	NA
AL	ALB	008	Albania	Republic of Albania	NA

We can also inspect the data dictionary contained in B.

exhibitSlot(B, "Ready4useDyad_r4", type_1L_chr = "dict", scroll_box_args_ls = list(width = "100%"))

Data Dictionary
Variable	Category	Description	Class
Alpha_2	A2	Alpabetical country code (two letters)	character
Alpha_3	A3	Alpabetical country code (three letters)	character
Numeric	N	Numeric country code	character
Name	Country	Country name	character
Official_name	Official	Country name (official)	character
Common_name	Common	Country name (common alternative)	character

We can now specifying both the concept (from the “Category” column of the data dictionary) that specifies allowable values for our target variable and all concepts we plan to use for fuzzy logic matching (described below).

B@label_1L_chr <- "Country"
B@include_chr <- c("Country", "Official","Common","A3","A2")

We now add B to X.

X <- renew(X, B, what_1L_chr = "standards")

Compare variable of interest values from seed and standards dataset.

To identify any disparities between the variable of interest in our seed and standards datasets we can use the ratify method. Supplying the value “identity” ensures that the output will differ from input only in the slot reserved for results.

X <- ratify(X, new_val_xx = "identity")

We can now identify the values from our seed dataset variable of interest that were not in our standard values.

X@results_ls$Country_Output_Validation$Invalid_Values

We can also identify standard values that were not present in the seed dataset variable of interest.

X@results_ls$Country_Output_Validation$Absent_Values
#>  [1] "Åland Islands"                                "Antarctica"                                  
#>  [3] "Bolivia, Plurinational State of"              "Bonaire, Sint Eustatius and Saba"            
#>  [5] "Bouvet Island"                                "British Indian Ocean Territory"              
#>  [7] "Brunei Darussalam"                            "Cabo Verde"                                  
#>  [9] "Christmas Island"                             "Cocos (Keeling) Islands"                     
#> [11] "Congo, The Democratic Republic of the"        "Côte d'Ivoire"                               
#> [13] "Curaçao"                                      "Czechia"                                     
#> [15] "Eswatini"                                     "Falkland Islands (Malvinas)"                 
#> [17] "French Southern Territories"                  "Guernsey"                                    
#> [19] "Heard Island and McDonald Islands"            "Holy See (Vatican City State)"               
#> [21] "Hong Kong"                                    "Iran, Islamic Republic of"                   
#> [23] "Korea, Democratic People's Republic of"       "Korea, Republic of"                          
#> [25] "Lao People's Democratic Republic"             "Macao"                                       
#> [27] "Micronesia, Federated States of"              "Moldova, Republic of"                        
#> [29] "Palestine, State of"                          "Réunion"                                     
#> [31] "Russian Federation"                           "Saint Barthélemy"                            
#> [33] "Saint Helena, Ascension and Tristan da Cunha" "Saint Martin (French part)"                  
#> [35] "Saint Vincent and the Grenadines"             "Sint Maarten (Dutch part)"                   
#> [37] "South Georgia and the South Sandwich Islands" "Syrian Arab Republic"                        
#> [39] "Taiwan, Province of China"                    "Tanzania, United Republic of"                
#> [41] "Timor-Leste"                                  "Turks and Caicos Islands"                    
#> [43] "United Kingdom"                               "United States"                               
#> [45] "United States Minor Outlying Islands"         "Venezuela, Bolivarian Republic of"           
#> [47] "Viet Nam"                                     "Virgin Islands, British"                     
#> [49] "Virgin Islands, U.S."

Standardise variable values

We can explore the extent to which we can use fuzzy logic to reconcile some of these discrepancies. To identify the types of fuzzy logic algorithms we could use, run the following command to explore the relevant part of the documentation from the stringdist library.

# Not run
# help("stringdist-metrics", package=stringdist)

In this case, we have chosen the Jaro, or Jaro-Winkler distance method (“jw”).

X <- renew(X, "jw", type_1L_chr = "slot", what_1L_chr = "logic") 
X <- ratify(X, new_val_xx = NULL)

This method will replace every previously invalid seed dataset variable value with the best available match identified by the selected fuzzy logic algorithm.

X@results_ls$Country_Output_Validation$Invalid_Values
#> character(0)

However, some of the replacements will be spurious as can be seen by inspecting the record of the replacements made.

X@results_ls$Country_Output_Correspondences
#> # A tibble: 41 × 2
#>    old_nms_chr               new_nms_chr                          
#>    <chr>                     <chr>                                
#>  1 Azores                    Timor-Leste                          
#>  2 Bolivia                   Bolivia, Plurinational State of      
#>  3 British Virgin Islands    Virgin Islands, British              
#>  4 Brunei                    Brunei Darussalam                    
#>  5 Canary Islands            Åland Islands                        
#>  6 Cape Verde                Cabo Verde                           
#>  7 Congo Democratic Republic Congo, The Democratic Republic of the
#>  8 Czech Republic            Czechia                              
#>  9 East Timor                Eswatini                             
#> 10 Easter Island             Christmas Island                     
#> # ℹ 31 more rows

For each of the incorrect correspondences, we will need to manually specify correct values. We can do this using the ready4show_correspondences sub-module.

# Not run
# a <- ready4show::renew.ready4show_correspondences(ready4show::ready4show_correspondences(), 
#         old_nms_chr = c("old_name_1", "old_name_2", "etc...."), new_nms_chr = c("new_name_1", "new_name_2", "etc...."))

The make_country_correspondences can be used as a shortcut for creating the alternative correspondences for this specific example.

a <- make_country_correspondences("cities")

We can inspect the values of this correspondence table.

exhibit(a, scroll_box_args_ls = list(width = "100%"))

Old name	New name
Azores	Portugal
Canary Islands	Spain
Easter Island	Chile
East Timor	Timor-Leste
Ivory Coast	Côte d'Ivoire
Kosovo	Kosovo
Madeira	Portugal
Netherlands Antilles	Bonaire, Sint Eustatius and Saba
Sicily	Italy
Vatican City	Holy See (Vatican City State)

When the ratify method was used to apply the fuzzy logic algorithm in a previous step, X was modified so that this logic is by default switched off for future calls to ratify. If we had created a new correspondence table that specified replacements for all invalid values, this would not be a problem. However, in this example we are only specifying correspondences where the fuzzy logic algorithm failed, so we need to again supply our desired fuzzy logic value.

X <- renew(X, "jw", type_1L_chr = "slot", what_1L_chr = "logic")

We now rerun our ratify method (which in this example will combine fuzzy logic with lookups from the manually created correspondences table).

X <- ratify(X, new_val_xx = a)

We once again inspect results.

Our correspondences table looks better.

X@results_ls$Country_Output_Correspondences
#> # A tibble: 41 × 2
#>    old_nms_chr               new_nms_chr                          
#>    <chr>                     <chr>                                
#>  1 Azores                    Portugal                             
#>  2 Bolivia                   Bolivia, Plurinational State of      
#>  3 British Virgin Islands    Virgin Islands, British              
#>  4 Brunei                    Brunei Darussalam                    
#>  5 Canary Islands            Spain                                
#>  6 Cape Verde                Cabo Verde                           
#>  7 Congo Democratic Republic Congo, The Democratic Republic of the
#>  8 Czech Republic            Czechia                              
#>  9 East Timor                Timor-Leste                          
#> 10 Easter Island             Chile                                
#> # ℹ 31 more rows

There is still a value that is not included in our standards.

X@results_ls$Country_Output_Validation$Invalid_Values
#> [1] "Kosovo"

We can rerun the ratify method to force the removal of any record that is not included in our standards dataset.

X <- renew(X, T, type_1L_chr = "slot", what_1L_chr = "force") 
X <- ratify(X, new_val_xx = "identity")

No invalid values remain.

X@results_ls$Country_Output_Validation$Invalid_Values
#> character(0)

However, there are also a some values from our standards dataset that are not represented in the results dataset values.

X@results_ls$Country_Output_Validation$Absent_Values
#>  [1] "Åland Islands"                                "Antarctica"                                  
#>  [3] "Bouvet Island"                                "British Indian Ocean Territory"              
#>  [5] "Christmas Island"                             "Cocos (Keeling) Islands"                     
#>  [7] "Curaçao"                                      "French Southern Territories"                 
#>  [9] "Heard Island and McDonald Islands"            "Hong Kong"                                   
#> [11] "Macao"                                        "Sint Maarten (Dutch part)"                   
#> [13] "South Georgia and the South Sandwich Islands" "United States Minor Outlying Islands"

Whether this is a problem or not depends on the intended purposes of the standardised dataset we are creating. We could choose to rerun the previous steps after making edits to either or both of the standards dataset (e.g. we could delete any superfluous, outdated or incorrect records or use an entirely new standards dataset) and seed dataset (e.g. adding new records or recategorising existing records so that there are corresponding values for every missing standard value). In this case we are going to assume that the above missing values are not a cause for concern for the valid use of our updated dataset for it intended purposes. We can now create a new object Y, using our results dataset’s Ready4useDyad module instance.

Y <- X@results_ls$Country_Output_Lookup

We can inspect the records for cases corresponding to capital cities from our new dataset.

renewSlot(Y,"ds_tb",Y@ds_tb %>% dplyr::filter(capital==1)) %>%
  renew(type_1L_chr = "label") %>%
  exhibit(scroll_box_args_ls = list(width = "100%"))

Dataset
City name	Country name	Population size	Latitude coordinate	Longitude coordinate	Is the nation's capital city
'Amman	Jordan	1303197	31.95	35.93	1
Abu Dhabi	United Arab Emirates	619316	24.48	54.37	1
Abuja	Nigeria	178462	9.18	7.17	1
Accra	Ghana	2029143	5.56	-0.20	1
Adamstown	Pitcairn	51	-25.05	-130.10	1
Addis Abeba	Ethiopia	2823167	9.03	38.74	1
Agana	Guam	1041	13.47	144.75	1
Algiers	Algeria	2029936	36.77	3.04	1
Alofi	Niue	627	-19.05	-169.92	1
Amsterdam	Netherlands	744159	52.37	4.89	1
Andorra la Vella	Andorra	20314	42.51	1.51	1
Ankara	Turkey	3579706	39.93	32.85	1
Antananarivo	Madagascar	1463754	-18.89	47.51	1
Apia	Samoa	40805	-13.83	-171.76	1
Asgabat	Turkmenistan	823013	37.95	58.38	1
Asmara	Eritrea	578860	15.33	38.94	1
Astana	Kazakhstan	351343	51.17	71.47	1
Asuncion	Paraguay	507574	-25.30	-57.63	1
Athens	Greece	725049	37.98	23.73	1
Avarua	Cook Islands	13645	-21.20	-159.76	1
Baghdad	Iraq	5753612	33.33	44.44	1
Bairiki	Kiribati	45982	1.33	172.99	1
Baku	Azerbaijan	1118725	40.39	49.86	1
Bamako	Mali	1342519	12.65	-7.99	1
Bandar Seri Begawan	Brunei Darussalam	67077	4.93	114.95	1
Bangkok	Thailand	4935988	13.73	100.50	1
Bangui	Central African Republic	547668	4.36	18.56	1
Banjul	Gambia	34388	13.46	-16.60	1
Basse-Terre	Guadeloupe	11298	16.00	-61.72	1
Basseterre	Saint Kitts and Nevis	12883	17.31	-62.73	1
Bayrut	Lebanon	1273440	33.88	35.50	1
Beijing	China	7602069	39.93	116.40	1
Belgrade	Serbia	1113589	44.83	20.50	1
Belmopan	Belize	14590	17.25	-88.79	1
Berlin	Germany	3378275	52.52	13.38	1
Bern	Switzerland	120596	46.95	7.44	1
Biskek	Kyrgyzstan	915625	42.87	74.57	1
Bissau	Guinea-Bissau	404119	11.87	-15.60	1
Bogota	Colombia	7235084	4.63	-74.09	1
Brasilia	Brazil	2260541	-15.78	-47.91	1
Bratislava	Slovakia	422452	48.16	17.13	1
Brazzaville	Congo	1326975	-4.25	15.26	1
Bridgetown	Barbados	98725	13.11	-59.61	1
Brussels	Belgium	1031925	50.83	4.33	1
Bucharest	Romania	1862930	44.44	26.10	1
Budapest	Hungary	1700019	47.51	19.08	1
Buenos Aires	Argentina	11595183	-34.61	-58.37	1
Bujumbura	Burundi	336561	-3.37	29.35	1
Cairo	Egypt	7836243	30.06	31.25	1
Canberra	Australia	324736	-35.31	149.13	1
Caracas	Venezuela, Bolivarian Republic of	1808937	10.54	-66.93	1
Castries	Saint Lucia	12904	14.03	-60.98	1
Cayenne	French Guiana	62926	4.92	-52.34	1
Charlotte Amalie	Virgin Islands, U.S.	10415	18.35	-64.94	1
Chisinau	Moldova, Republic of	623671	47.03	28.83	1
Cockburn Town	Turks and Caicos Islands	174	21.46	-71.14	1
Colombo	Sri Lanka	649496	6.93	79.85	1
Conakry	Guinea	1970382	9.55	-13.67	1
Copenhagen	Denmark	1091978	55.68	12.57	1
Dakar	Senegal	2406598	14.72	-17.48	1
Damascus	Syrian Arab Republic	1580909	33.50	36.32	1
Dhaka	Bangladesh	6724976	23.70	90.39	1
Dili	Timor-Leste	163305	-8.57	125.58	1
Dodoma	Tanzania, United Republic of	188150	-6.17	35.74	1
Doha	Qatar	351381	25.30	51.51	1
Douglas	Isle of Man	25621	54.15	-4.48	1
Dublin	Ireland	1030431	53.33	-6.25	1
Dushanbe	Tajikistan	538456	38.57	68.78	1
Dzaoudzi	Mayotte	14558	-12.77	45.25	1
Fakaofo	Tokelau	267	-9.38	-171.22	1
Fort-de-France	Martinique	89233	14.60	-61.08	1
Freetown	Sierra Leone	818709	8.49	-13.24	1
Gaborone	Botswana	214412	-24.65	25.91	1
George Town	Cayman Islands	30570	19.28	-81.39	1
Georgetown	Guyana	236878	6.79	-58.16	1
Gibraltar	Gibraltar	26404	36.14	-5.35	1
Guatemala	Guatemala	1010253	14.63	-90.55	1
Ha Noi	Viet Nam	1452055	21.03	105.84	1
Hamilton	Bermuda	889	32.30	-64.79	1
Harare	Zimbabwe	1575127	-17.82	31.05	1
Havanna	Cuba	2163132	23.13	-82.39	1
Helsinki	Finland	558341	60.17	24.94	1
Honiara	Solomon Islands	57410	-9.43	159.91	1
Islamabad	Pakistan	794431	33.72	73.06	1
Jakarta	Indonesia	8556798	-6.18	106.83	1
Jamestown	Saint Helena, Ascension and Tristan da Cunha	603	-15.92	-5.71	1
Jerusalem	Israel	731731	31.78	35.22	1
Jibuti	Djibouti	633884	11.56	43.15	1
Kabul	Afghanistan	3120963	34.53	69.17	1
Kampala	Uganda	1403619	0.32	32.58	1
Kathmandu	Nepal	822930	27.71	85.31	1
Khartoum	Sudan	2090001	15.58	32.52	1
Kiev	Ukraine	2491404	50.43	30.52	1
Kigali	Rwanda	800003	-1.94	30.06	1
Kingston	Jamaica	585300	17.99	-76.80	1
Kingston	Norfolk Island	890	-29.03	168.05	1
Kingstown	Saint Vincent and the Grenadines	18160	13.16	-61.23	1
Kinshasa	Congo, The Democratic Republic of the	8096254	-4.31	15.32	1
Koror	Palau	11458	7.35	134.51	1
Kuala Lumpur	Malaysia	1482359	3.16	101.71	1
Libreville	Gabon	591356	0.39	9.45	1
Lilongwe	Malawi	683477	-13.97	33.80	1
Lima	Peru	7857121	-12.07	-77.05	1
Lisbon	Portugal	508209	38.72	-9.14	1
Ljubljana	Slovenia	254188	46.06	14.51	1
Lome	Togo	737751	6.17	1.35	1
London	United Kingdom	7489022	51.52	-0.10	1
Longyearbyen	Svalbard and Jan Mayen	1263	78.21	15.61	1
Luanda	Angola	2875277	-8.82	13.24	1
Lusaka	Zambia	1306577	-15.42	28.29	1
Luxemburg	Luxembourg	76380	49.62	6.12	1
Madrid	Spain	3146804	40.42	-3.71	1
Malabo	Equatorial Guinea	161409	3.74	8.79	1
Male	Maldives	87154	4.17	73.50	1
Managua	Nicaragua	990417	12.15	-86.27	1
Manama	Bahrain	147894	26.21	50.58	1
Manila	Philippines	10546511	14.62	120.97	1
Maputo	Mozambique	1220167	-25.95	32.57	1
Maseru	Lesotho	116268	-29.31	27.49	1
Mata'utu	Wallis and Futuna	1310	-13.28	-176.13	1
Mbabane	Eswatini	78740	-26.32	31.14	1
Mexico City	Mexico	8659409	19.43	-99.14	1
Minsk	Belarus	1747482	53.91	27.55	1
Mogadishu	Somalia	2723378	2.05	45.33	1
Monaco-Ville	Monaco	975	43.74	7.42	1
Monrovia	Liberia	954458	6.31	-10.80	1
Montevideo	Uruguay	1271664	-34.87	-56.17	1
Moroni	Comoros	43704	-11.74	43.23	1
Moscow	Russian Federation	10472629	55.75	37.62	1
Muscat	Oman	24122	23.61	58.54	1
N'Djamena	Chad	737281	12.11	15.05	1
Nairobi	Kenya	2864667	-1.29	36.82	1
Nassau	Bahamas	231519	25.06	-77.33	1
Ni Dilli	India	321883	28.60	77.22	1
Niamey	Niger	801297	13.52	2.12	1
Nicosia	Cyprus	202488	35.16	33.38	1
Nicosia	Cyprus	42372	35.18	33.37	1
Nouakchott	Mauritania	731242	18.09	-15.98	1
Noumea	New Caledonia	94751	-22.27	166.44	1
Nuku'alofa	Tonga	23733	-21.14	-175.22	1
Nuuk	Greenland	15243	64.18	-51.73	1
Oranjestad	Aruba	30710	12.53	-70.03	1
Oslo	Norway	821445	59.91	10.75	1
Ottawa	Canada	885542	45.42	-75.71	1
Ouagadougou	Burkina Faso	1119775	12.37	-1.53	1
Pago Pago	American Samoa	4180	-14.24	-170.72	1
Palikir	Micronesia, Federated States of	4552	6.92	158.16	1
Panama	Panama	406070	8.97	-79.53	1
Papeete	French Polynesia	26400	-17.52	-149.56	1
Paramaribo	Suriname	224925	5.85	-55.20	1
Paris	France	2141839	48.86	2.34	1
Phnum Penh	Cambodia	1673131	11.57	104.92	1
Port Louis	Mauritius	156760	-20.17	57.51	1
Port Moresby	Papua New Guinea	289861	-9.48	147.18	1
Port Stanley	Falkland Islands (Malvinas)	2269	-51.70	-57.82	1
Port of Spain	Trinidad and Tobago	49764	10.66	-61.51	1
Port-au-Prince	Haiti	1277104	18.54	-72.34	1
Porto Novo	Benin	238199	6.48	2.63	1
Prague	Czechia	1168374	50.08	14.43	1
Praia	Cabo Verde	117342	14.93	-23.54	1
Pretoria	South Africa	1687779	-25.73	28.22	1
Pyongyang	Korea, Democratic People's Republic of	2992272	39.02	125.75	1
Quito	Ecuador	1399814	-0.19	-78.50	1
Rabat	Morocco	1688738	34.02	-6.84	1
Rangoon	Myanmar	4572948	16.79	96.15	1
Reykjavik	Iceland	114576	64.14	-21.92	1
Riga	Latvia	738386	56.97	24.13	1
Rita	Marshall Islands	21270	7.12	171.06	1
Riyadh	Saudi Arabia	4328067	24.65	46.77	1
Road Town	Virgin Islands, British	8613	18.43	-64.63	1
Rome	Italy	2561181	41.89	12.50	1
Roseau	Dominica	16577	15.30	-61.39	1
Saint George's	Grenada	4315	12.06	-61.74	1
Saint Helier	Jersey	28910	49.19	-2.11	1
Saint John's	Antigua and Barbuda	25321	17.11	-61.85	1
Saint Peter Port	Guernsey	16702	49.47	-2.55	1
Saint-Denis	Réunion	137787	-20.87	55.46	1
Saint-Pierre	Saint Pierre and Miquelon	6254	46.79	-56.18	1
San Jose	Costa Rica	32187	10.97	-85.13	1
San Jose	Costa Rica	339588	9.93	-84.08	1
San Juan	Puerto Rico	417154	18.44	-66.13	1
San Marino	San Marino	4624	43.94	12.43	1
San Salvador	El Salvador	534409	13.69	-89.19	1
San'a	Yemen	1921589	15.38	44.21	1
Santiago	Chile	4893495	-33.46	-70.64	1
Santo Domingo	Dominican Republic	2253437	18.48	-69.91	1
Sao Tome	Sao Tome and Principe	63772	0.37	6.73	1
Sarajevo	Bosnia and Herzegovina	737350	43.85	18.38	1
Singapore	Singapore	3601745	1.30	103.85	1
Skopje	North Macedonia	477493	42.00	21.47	1
Sofia	Bulgaria	1166143	42.69	23.31	1
Seoul	Korea, Republic of	10409345	37.56	126.99	1
Stockholm	Sweden	1260712	59.33	18.07	1
Sucre	Bolivia, Plurinational State of	232669	-19.06	-65.26	1
Susupe	Northern Mariana Islands	2402	15.14	145.70	1
Taipei	Taiwan, Province of China	2491662	25.02	121.45	1
Tallinn	Estonia	392386	59.44	24.74	1
Tashkent	Uzbekistan	1967879	41.31	69.30	1
Tbilisi	Georgia	1038343	41.72	44.79	1
Tegucigalpa	Honduras	872403	14.09	-87.22	1
Tehran	Iran, Islamic Republic of	7160094	35.67	51.43	1
The Valley	Anguilla	1435	18.22	-63.05	1
Thimphu	Bhutan	74175	27.48	89.70	1
Tirana	Albania	380403	41.33	19.82	1
Tokyo	Japan	8372440	35.67	139.77	1
Torshavn	Faroe Islands	13313	62.03	-6.80	1
Tripoli	Libya	1164634	32.87	13.18	1
Tunis	Tunisia	693294	36.84	10.22	1
Ulaanbaatar	Mongolia	862842	47.93	106.91	1
Vaduz	Liechtenstein	5248	47.14	9.53	1
Vaiaku	Tuvalu	4835	-8.52	179.20	1
Valletta	Malta	6748	35.91	14.52	1
Vatican City	Holy See (Vatican City State)	767	41.90	12.46	1
Victoria	Seychelles	22611	-4.62	55.45	1
Vienna	Austria	1570976	48.22	16.37	1
Vientiane	Lao People's Democratic Republic	199863	17.97	102.61	1
Vila	Vanuatu	37141	-17.74	168.31	1
Vilnius	Lithuania	542014	54.70	25.27	1
Warsaw	Poland	1634441	52.26	21.02	1
Washington	United States	548359	38.91	-77.02	1
Wellington	New Zealand	182254	-41.28	174.78	1
Willemstad	Bonaire, Sint Eustatius and Saba	98339	12.10	-68.93	1
Windhoek	Namibia	277349	-22.56	17.09	1
Yamoussoukro	Côte d'Ivoire	200103	6.82	-5.28	1
Yaounde	Cameroon	1344617	3.87	11.52	1
Yaren	Nauru	4587	-0.55	166.91	1
Yerevan	Armenia	1090537	40.17	44.52	1
Zagreb	Croatia	700717	45.80	15.97	1
al-'Ayun	Western Sahara	188084	27.16	-13.20	1
al-Kuwayt	Kuwait	63596	29.38	47.99	1

Docs: Add a data dictionary to a dataset

Wed, 28 Jan 2026 00:00:00 +0000

This below section renders a vignette article from the ready4use library. You can use the following links to:

view the vignette on the library website (adds useful hyperlinks to code blocks)
view the source file from that article, and;
edit its contents (requires a GitHub account).

Note: This vignette is illustrated with fake data. The dataset explored in this example should not be used to inform decision-making.

library(ready4)
library(ready4use)

ready4use includes a number of tools for labeling health economic model data and forms part of the ready4 framework.

Create a dataset-dictionary pair

A data dictionary contains useful metadata about a dataset. To illustrate this point, we can ingest examples of a toy (fake) dataset and its data-dictionary.

objects_ls <- Ready4useRepos(dv_nm_1L_chr = "fakes",
                    dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/HJXYKQ",
                    dv_server_1L_chr = "dataverse.harvard.edu") %>%
  ingest(metadata_1L_lgl = F)

Importantly (and a requirement for subsequent steps), the data dictionary we ingest is a ready4use_dictionary object.

class(objects_ls$eq5d_ds_dict)
#> [1] "ready4use_dictionary" "ready4_dictionary"    "tbl_df"               "tbl"                  "data.frame"

We can now pair the data dictionary with its dataset in a new object X, a Ready4useDyad.

X <- Ready4useDyad(ds_tb = objects_ls$eq5d_ds_tb,
                   dictionary_r3 = objects_ls$eq5d_ds_dict)

Inspect data

We can inspect X by printing selected information about it to console using the exhibit method. If we only wish to see the first or last few records, we can pass “head” or “tail” to the display_1L_chr argument.

 exhibit(X,
         display_1L_chr = "head",
         scroll_box_args_ls = list(width = "100%"))

Dataset
uid	Timepoint	data_collection_dtm	d_age	Gender	d_sex_birth_s	d_sexual_ori_s	d_relation_s	d_ATSI	CALD	Region	d_studying_working	eq5dq_MO	eq5dq_SC	eq5dq_UA	eq5dq_PD	eq5dq_AD	K10_int	Psych_well_int
1	BL	2019-10-22	14	Male	Male	Heterosexual	In a relationship	No	No	Metro	Not studying or working	1	1	1	1	2	11	87
2	BL	2019-10-17	19	Female	Female	Heterosexual	In a relationship	Yes	Yes	Regional	Studying only	1	2	1	1	1	14	65
2	FUP	2020-02-14	19	Female	Female	Heterosexual	In a relationship	Yes	Yes	Regional	Studying only	3	1	1	1	1	10	71
3	BL	2020-02-15	21	Female	Female	Other	Not in a relationship	NA	NA	Metro	Studying only	1	1	3	1	1	13	74
3	FUP	2020-06-14	21	Female	Female	Other	Not in a relationship	NA	NA	Metro	Studying only	1	1	2	1	1	10	64
4	BL	2019-12-14	12	Female	Female	Heterosexual	In a relationship	Yes	Yes	Metro	Not studying or working	1	1	1	3	1	18	40

The dataset may be more meaningful if variables are labelled using the descriptive information from the data dictionary. This can be accomplished using the renew method.

X <- renew(X,
           type_1L_chr = "label")

exhibit(X,
        display_1L_chr = "head",
         scroll_box_args_ls = list(width = "100%"))

Dataset
Unique identifier	Data collection round	Date of data collection	Age	Gender (grouped)	Sex at birth	Sexual orientation	Relationship status	Aboriginal or Torres Strait Islander	Culturally And Linguistically Diverse	Region of residence (metropolitan or regional)	Education and employment status	EQ5D - Mobility domain score	EQ5D - Self-Care domain score	EQ5D - Usual Activities domain score	EQ5D - Pain / Discomfort domain score	EQ5D - Anxiety / Depression domain score	Kessler Psychological Distress - 10 Item Total Score	Overall Wellbeing Measure (Winefield et al. 2012)
1	BL	2019-10-22	14	Male	Male	Heterosexual	In a relationship	No	No	Metro	Not studying or working	1	1	1	1	2	11	87
2	BL	2019-10-17	19	Female	Female	Heterosexual	In a relationship	Yes	Yes	Regional	Studying only	1	2	1	1	1	14	65
2	FUP	2020-02-14	19	Female	Female	Heterosexual	In a relationship	Yes	Yes	Regional	Studying only	3	1	1	1	1	10	71
3	BL	2020-02-15	21	Female	Female	Other	Not in a relationship	NA	NA	Metro	Studying only	1	1	3	1	1	13	74
3	FUP	2020-06-14	21	Female	Female	Other	Not in a relationship	NA	NA	Metro	Studying only	1	1	2	1	1	10	64
4	BL	2019-12-14	12	Female	Female	Heterosexual	In a relationship	Yes	Yes	Metro	Not studying or working	1	1	1	3	1	18	40

To remove dataset labels, use the renew method with “unlabel” passed to the type_1L_chr argument.

X <- renew(X,
           type_1L_chr = "unlabel")

By default, the exhibit method will print the dataset part of the Ready4useDyad instance. To inspect the data dictionary, pass “dict” to the type_1L_chr argument.

exhibit(X,
        display_1L_chr = "head",
        type_1L_chr = "dict",
        scroll_box_args_ls = list(width = "100%"))

Data Dictionary
Variable	Category	Description	Class
CALD	demographic	Culturally And Linguistically Diverse	factor
d_age	demographic	age	integer
d_ATSI	demographic	Aboriginal or Torres Strait Islander	character
d_relation_s	demographic	relationship status	character
d_sex_birth_s	demographic	sex at birth	character
d_sexual_ori_s	demographic	sexual orientation	factor

Docs: Standardise Variable Values With Lookup Codes

Thu, 30 Nov 2023 00:00:00 +0000

This below section renders a vignette article from the costly library. You can use the following links to:

view the vignette on the library website (adds useful hyperlinks to code blocks)
view the source file from that article, and;
edit its contents (requires a GitHub account).

library(ready4)
library(ready4use)
library(costly)

Note. Parts of the workflow described in this article are common to steps explained in more detail in the article outlining the workflow using fuzzy logic and correspondence tables.

In brief

The steps described and explained in this vignette can also be (more succinctly) accomplished with the following code.

X <- CostlyCountries()
X <- renew(X,
           new_val_xx = add_default_currency_seed(X@CostlySeed_r4, include_1L_chr = "Country"), 
           what_1L_chr = "seed")
X <- renew(X, "jw", type_1L_chr = "slot", what_1L_chr = "logic") 
X <- renew(X, new_val_xx = make_country_correspondences("currencies"), what_1L_chr = "correspondences") 
X <- renew(X, T, type_1L_chr = "slot", what_1L_chr = "force") 
X <- ratify(X)
Y <- CostlyCurrencies()
Y <- renew(Y, new_val_xx = add_default_currency_seed(Y@CostlySeed_r4,
                                                     Ready4useDyad_r4 = X@results_ls$Country_Output_Lookup), 
           what_1L_chr = "seed")
Y <- ratify(Y, type_1L_chr = "Lookup")
Y <- renew(Y, T, type_1L_chr = "slot", what_1L_chr = "force") 
Y <- ratify(Y, type_1L_chr = "Lookup")

Create project

We begin by creating X, a CostlyCorrespondences module instance.

X <- CostlyCorrespondences()

Supply seed dataset

We next create a CostlySeed module instance that includes a dataset containing our variable of interest (in this case, countries). The dataset needs to be paired with a dataset dictionary using the Ready4useDyad module from the ready4use R library. You can supply a custom standards dataset (a tibble), dictionary (a ready4use_dictionary) and the concept represented by our variable of interest using a command of the following format.

# Not run
# A <- CostlySeed(Ready4useDyad_r4 = Ready4useDyad(ds_tb = tibble::tibble(), dictionary_r3 = ready4use_dictionary()), include_chr = c("Country"), label_1L_chr = "Country")

A <- CostlySeed() %>% add_default_currency_seed()

We now add A to a new CostlyCorrespondences module instance Y, which we use to standardise the country concept variable using a fuzzy logic And correspondence tables workflow.

A@include_chr <- A@label_1L_chr <- "Country"
Y <- CostlyCountries(CostlySeed_r4 = A) %>%
  renew("jw", type_1L_chr = "slot", what_1L_chr = "logic") %>%
  renew(new_val_xx = make_country_correspondences("currencies"), what_1L_chr = "correspondences") %>%
  renew(T, type_1L_chr = "slot", what_1L_chr = "force") %>%
  ratify()

We now update X with the results Ready4useDyad from Y (a seed dataset for which country names have been standardised).

X <- renew(X, new_val_xx = CostlySeed(Ready4useDyad_r4 = Y@results_ls$Country_Output_Lookup), what_1L_chr = "seed") #

We can now inspect the first few records from our labelled seed dataset.

renewSlot(X, "CostlySeed_r4@Ready4useDyad_r4", type_1L_chr = "label") %>%
exhibitSlot("CostlySeed_r4@Ready4useDyad_r4", display_1L_chr = "head", scroll_box_args_ls = list(width = "100%"))

Dataset
Country name	Currency name	Currency symbol	Currency alphabetical ISO code (three letter)	Currency's fractional unit	Number of fractional units in basic unit
Afghanistan	Afghan afghani	؋‎	AFN	Pul	100
Albania	Albanian lek	Lek	ALL	Qintar	100
Algeria	Algerian dinar	DA	DZD	Santeem	100
Andorra	Euro	€	EUR	Cent	100
Angola	Angolan kwanza	Kz	AOA	Cêntimo	100
Anguilla	Eastern Caribbean dollar	\$	XCD	Cent	100

We can also inspect the seed dataset’s dictionary.

exhibitSlot(X, "CostlySeed_r4@Ready4useDyad_r4", type_1L_chr = "dict", scroll_box_args_ls = list(width = "100%"))

Data Dictionary
Variable	Category	Description	Class
State or territory\[1\]	Country	Country name	character
Currency\[1\]\[2\]	Currency	Currency name	character
Symbol\[D\] orAbbrev.\[3\]	Symbol	Currency symbol	character
ISO code\[2\]	A3	Currency alphabetical ISO code (three letter)	character
Fractionalunit	Fractional	Currency's fractional unit	character
Numberto basic	Number	Number of fractional units in basic unit	character

We specify the seed dataset concept that we are looking to standardise and the concept that we will use to lookup replacement values from the standards dataset.

X@CostlySeed_r4@label_1L_chr <- "Currency"
X@CostlySeed_r4@match_1L_chr <- "A3"

Specify standards

We can now create B, a CostlyStandards module instance that includes a dataset specifying the complete list of allowable variable values. In many cases using the ISO_4217 dataset from the ISOcodes library will be the optimal source of standardised names for currencies. Using the add_currency_standards function will pair this dataset with a dictionary.

B <- CostlyStandards(Ready4useDyad_r4 = Ready4useDyad() %>% add_currency_standards())

We can inspect the first few cases of the labelled version of the standards dataset in B.

renewSlot(B, "Ready4useDyad_r4", type_1L_chr = "label") %>% 
  exhibitSlot("Ready4useDyad_r4", display_1L_chr = "head", scroll_box_args_ls = list(width = "100%"))

Dataset
Alpabetical currency code (three letters)	Numeric currency code	Currency name
AED	784	UAE Dirham
AFN	971	Afghani
ALL	008	Lek
AMD	051	Armenian Dram
ANG	532	Netherlands Antillean Guilder
AOA	973	Kwanza

We can also inspect the data dictionary contained in B.

exhibitSlot(B, "Ready4useDyad_r4", type_1L_chr = "dict", scroll_box_args_ls = list(width = "100%"))

Data Dictionary
Variable	Category	Description	Class
Letter	A3	Alpabetical currency code (three letters)	character
Numeric	N	Numeric currency code	character
Currency	Currency	Currency name	character

We can now specifying both the concept (“Currency”) that specifies allowable values for our target variable and the concepts we plan to use for lookup matching (described below).

#B@include_chr <- c("Currency", "Letter")
B@label_1L_chr <- "Currency"
B@match_1L_chr <- "A3"

We now add B to X.

X <- renew(X, B, what_1L_chr = "standards")

Compare variable of interest values from seed and standards dataset.

Currently, the majority of our currency names need to be standardised. In many cases this may be due to something as simple as the use of lower case.

X <- ratify(X, new_val_xx = "identity")
X@results_ls$Currency_Output_Validation$Invalid_Values
#>   [1] "Afghan afghani"                          "Albanian lek"                            "Algerian dinar"                         
#>   [4] "Angolan kwanza"                          "Argentine peso"                          "Armenian dram"                          
#>   [7] "Aruban florin"                           "Australian dollar"                       "Azerbaijani manat"                      
#>  [10] "Bahamian dollar"                         "Bahraini dinar"                          "Bangladeshi taka"                       
#>  [13] "Barbadian dollar"                        "Belarusian ruble"                        "Belize dollar"                          
#>  [16] "Bermudian dollar"                        "Bhutanese ngultrum"                      "Bitcoin[4] (as legal tender)"           
#>  [19] "Bolivian boliviano"                      "Bosnia and Herzegovina convertible mark" "Botswana pula"                          
#>  [22] "Brazilian real"                          "Brunei dollar"                           "Bulgarian lev"                          
#>  [25] "Burmese kyat"                            "Burundian franc"                         "Cambodian riel"                         
#>  [28] "Canadian dollar"                         "Cape Verdean escudo"                     "Cayman Islands dollar"                  
#>  [31] "Central African CFA franc"               "CFP franc"                               "Chilean peso"                           
#>  [34] "Colombian peso"                          "Comorian franc"                          "Congolese franc"                        
#>  [37] "Cook Islands dollar"                     "Costa Rican colón"                       "Cuban peso"                             
#>  [40] "Czech koruna"                            "Danish krone"                            "Djiboutian franc"                       
#>  [43] "Dominican peso"                          "Eastern Caribbean dollar"                "Egyptian pound"                         
#>  [46] "Eritrean nakfa"                          "Ethiopian birr"                          "Falkland Islands pound"                 
#>  [49] "Faroese króna"                           "Fijian dollar"                           "Gambian dalasi"                         
#>  [52] "Georgian lari"                           "Ghanaian cedi"                           "Gibraltar pound"                        
#>  [55] "Guatemalan quetzal"                      "Guernsey pound"                          "Guinean franc"                          
#>  [58] "Guyanese dollar"                         "Haitian gourde"                          "Honduran lempira"                       
#>  [61] "Hong Kong dollar"                        "Hungarian forint"                        "Icelandic króna"                        
#>  [64] "Indian rupee"                            "Indonesian rupiah"                       "Iranian rial"                           
#>  [67] "Iraqi dinar"                             "Israeli new shekel"                      "Jamaican dollar"                        
#>  [70] "Japanese yen"                            "Jersey pound"                            "Jordanian dinar"                        
#>  [73] "Kazakhstani tenge"                       "Kenyan shilling"                         "Kiribati dollar[E]"                     
#>  [76] "Kuwaiti dinar"                           "Kyrgyz som"                              "Lao kip"                                
#>  [79] "Lebanese pound"                          "Lesotho loti"                            "Liberian dollar"                        
#>  [82] "Libyan dinar"                            "Macanese pataca"                         "Macedonian denar"                       
#>  [85] "Malagasy ariary"                         "Malawian kwacha"                         "Malaysian ringgit"                      
#>  [88] "Maldivian rufiyaa"                       "Manx pound"                              "Mauritanian ouguiya"                    
#>  [91] "Mauritian rupee"                         "Mexican peso"                            "Moldovan leu"                           
#>  [94] "Mongolian tögrög"                        "Moroccan dirham"                         "Mozambican metical"                     
#>  [97] "Namibian dollar"                         "Nepalese rupee"                          "Netherlands Antillean guilder"          
#> [100] "New Taiwan dollar"                       "New Zealand dollar"                      "Nicaraguan córdoba"                     
#> [103] "Nigerian naira"                          "Niue dollar[E]"                          "North Korean won"                       
#> [106] "Norwegian krone"                         "Omani rial"                              "Pakistani rupee"                        
#> [109] "Panamanian balboa"                       "Papua New Guinean kina"                  "Paraguayan guaraní"                     
#> [112] "Peruvian sol"                            "Philippine peso"                         "Pitcairn Islands dollar[E]"             
#> [115] "Polish złoty"                            "Qatari riyal"                            "Renminbi"                               
#> [118] "Romanian leu"                            "Russian ruble"                           "Rwandan franc"                          
#> [121] "Sahrawi peseta"                          "Saint Helena pound"                      "Samoan tālā"                            
#> [124] "São Tomé and Príncipe dobra"             "Saudi riyal"                             "Serbian dinar"                          
#> [127] "Seychellois rupee"                       "Sierra Leonean leone"                    "Singapore dollar"                       
#> [130] "Solomon Islands dollar"                  "Somali shilling"                         "South African rand"                     
#> [133] "South Korean won"                        "South Sudanese pound"                    "Sri Lankan rupee"                       
#> [136] "Sterling"                                "Sudanese pound"                          "Surinamese dollar"                      
#> [139] "Swazi lilangeni"                         "Swedish krona"                           "Swiss franc"                            
#> [142] "Syrian pound"                            "Tajikistani somoni"                      "Tanzanian shilling"                     
#> [145] "Thai baht"                               "Tongan paʻanga[K]"                       "Trinidad and Tobago dollar"             
#> [148] "Tunisian dinar"                          "Turkish lira"                            "Turkmenistani manat"                    
#> [151] "Tuvaluan dollar"                         "Ugandan shilling"                        "Ukrainian hryvnia"                      
#> [154] "United Arab Emirates dirham"             "United States dollar"                    "United States dollar[F]"                
#> [157] "Uruguayan peso"                          "Uzbekistani sum"                         "Vanuatu vatu"                           
#> [160] "Venezuelan digital bolívar"              "Venezuelan sovereign bolívar"            "Vietnamese đồng"                        
#> [163] "West African CFA franc"                  "Yemeni rial"                             "Zambian kwacha"                         
#> [166] "Zimbabwean dollar"

Standardised currency names not currently present in our seed dataset are as follows.

X@results_ls$Currency_Output_Validation$Absent_Values
#>   [1] "ADB Unit of Account"                                              
#>   [2] "Afghani"                                                          
#>   [3] "Algerian Dinar"                                                   
#>   [4] "Argentine Peso"                                                   
#>   [5] "Armenian Dram"                                                    
#>   [6] "Aruban Florin"                                                    
#>   [7] "Australian Dollar"                                                
#>   [8] "Azerbaijan Manat"                                                 
#>   [9] "Bahamian Dollar"                                                  
#>  [10] "Bahraini Dinar"                                                   
#>  [11] "Baht"                                                             
#>  [12] "Balboa"                                                           
#>  [13] "Barbados Dollar"                                                  
#>  [14] "Belarusian Ruble"                                                 
#>  [15] "Belize Dollar"                                                    
#>  [16] "Bermudian Dollar"                                                 
#>  [17] "Bolívar Soberano"                                                 
#>  [18] "Boliviano"                                                        
#>  [19] "Bond Markets Unit European Composite Unit (EURCO)"                
#>  [20] "Bond Markets Unit European Monetary Unit (E.M.U.-6)"              
#>  [21] "Bond Markets Unit European Unit of Account 17 (E.U.A.-17)"        
#>  [22] "Bond Markets Unit European Unit of Account 9 (E.U.A.-9)"          
#>  [23] "Brazilian Real"                                                   
#>  [24] "Brunei Dollar"                                                    
#>  [25] "Bulgarian Lev"                                                    
#>  [26] "Burundi Franc"                                                    
#>  [27] "Cabo Verde Escudo"                                                
#>  [28] "Cayman Islands Dollar"                                            
#>  [29] "CFA Franc BCEAO"                                                  
#>  [30] "CFA Franc BEAC"                                                   
#>  [31] "CFP Franc"                                                        
#>  [32] "Chilean Peso"                                                     
#>  [33] "Codes specifically reserved for testing purposes"                 
#>  [34] "Colombian Peso"                                                   
#>  [35] "Comorian Franc"                                                   
#>  [36] "Congolese Franc"                                                  
#>  [37] "Convertible Mark"                                                 
#>  [38] "Cordoba Oro"                                                      
#>  [39] "Costa Rican Colon"                                                
#>  [40] "Cuban Peso"                                                       
#>  [41] "Czech Koruna"                                                     
#>  [42] "Dalasi"                                                           
#>  [43] "Danish Krone"                                                     
#>  [44] "Denar"                                                            
#>  [45] "Djibouti Franc"                                                   
#>  [46] "Dobra"                                                            
#>  [47] "Dominican Peso"                                                   
#>  [48] "Dong"                                                             
#>  [49] "East Caribbean Dollar"                                            
#>  [50] "Egyptian Pound"                                                   
#>  [51] "El Salvador Colon"                                                
#>  [52] "Ethiopian Birr"                                                   
#>  [53] "Falkland Islands Pound"                                           
#>  [54] "Fiji Dollar"                                                      
#>  [55] "Forint"                                                           
#>  [56] "Ghana Cedi"                                                       
#>  [57] "Gibraltar Pound"                                                  
#>  [58] "Gold"                                                             
#>  [59] "Gourde"                                                           
#>  [60] "Guarani"                                                          
#>  [61] "Guinean Franc"                                                    
#>  [62] "Guyana Dollar"                                                    
#>  [63] "Hong Kong Dollar"                                                 
#>  [64] "Hryvnia"                                                          
#>  [65] "Iceland Krona"                                                    
#>  [66] "Indian Rupee"                                                     
#>  [67] "Iranian Rial"                                                     
#>  [68] "Iraqi Dinar"                                                      
#>  [69] "Jamaican Dollar"                                                  
#>  [70] "Jordanian Dinar"                                                  
#>  [71] "Kenyan Shilling"                                                  
#>  [72] "Kina"                                                             
#>  [73] "Kuna"                                                             
#>  [74] "Kuwaiti Dinar"                                                    
#>  [75] "Kwanza"                                                           
#>  [76] "Kyat"                                                             
#>  [77] "Lao Kip"                                                          
#>  [78] "Lari"                                                             
#>  [79] "Lebanese Pound"                                                   
#>  [80] "Lek"                                                              
#>  [81] "Lempira"                                                          
#>  [82] "Leone"                                                            
#>  [83] "Liberian Dollar"                                                  
#>  [84] "Libyan Dinar"                                                     
#>  [85] "Lilangeni"                                                        
#>  [86] "Loti"                                                             
#>  [87] "Malagasy Ariary"                                                  
#>  [88] "Malawi Kwacha"                                                    
#>  [89] "Malaysian Ringgit"                                                
#>  [90] "Mauritius Rupee"                                                  
#>  [91] "Mexican Peso"                                                     
#>  [92] "Mexican Unidad de Inversion (UDI)"                                
#>  [93] "Moldovan Leu"                                                     
#>  [94] "Moroccan Dirham"                                                  
#>  [95] "Mozambique Metical"                                               
#>  [96] "Mvdol"                                                            
#>  [97] "Naira"                                                            
#>  [98] "Nakfa"                                                            
#>  [99] "Namibia Dollar"                                                   
#> [100] "Nepalese Rupee"                                                   
#> [101] "Netherlands Antillean Guilder"                                    
#> [102] "New Israeli Sheqel"                                               
#> [103] "New Taiwan Dollar"                                                
#> [104] "New Zealand Dollar"                                               
#> [105] "Ngultrum"                                                         
#> [106] "North Korean Won"                                                 
#> [107] "Norwegian Krone"                                                  
#> [108] "Ouguiya"                                                          
#> [109] "Pa’anga"                                                          
#> [110] "Pakistan Rupee"                                                   
#> [111] "Palladium"                                                        
#> [112] "Pataca"                                                           
#> [113] "Peso Convertible"                                                 
#> [114] "Peso Uruguayo"                                                    
#> [115] "Philippine Peso"                                                  
#> [116] "Platinum"                                                         
#> [117] "Pound Sterling"                                                   
#> [118] "Pula"                                                             
#> [119] "Qatari Rial"                                                      
#> [120] "Quetzal"                                                          
#> [121] "Rand"                                                             
#> [122] "Rial Omani"                                                       
#> [123] "Riel"                                                             
#> [124] "Romanian Leu"                                                     
#> [125] "Rufiyaa"                                                          
#> [126] "Rupiah"                                                           
#> [127] "Russian Ruble"                                                    
#> [128] "Rwanda Franc"                                                     
#> [129] "Saint Helena Pound"                                               
#> [130] "Saudi Riyal"                                                      
#> [131] "SDR (Special Drawing Right)"                                      
#> [132] "Serbian Dinar"                                                    
#> [133] "Seychelles Rupee"                                                 
#> [134] "Silver"                                                           
#> [135] "Singapore Dollar"                                                 
#> [136] "Sol"                                                              
#> [137] "Solomon Islands Dollar"                                           
#> [138] "Som"                                                              
#> [139] "Somali Shilling"                                                  
#> [140] "Somoni"                                                           
#> [141] "South Sudanese Pound"                                             
#> [142] "Sri Lanka Rupee"                                                  
#> [143] "Sucre"                                                            
#> [144] "Sudanese Pound"                                                   
#> [145] "Surinam Dollar"                                                   
#> [146] "Swedish Krona"                                                    
#> [147] "Swiss Franc"                                                      
#> [148] "Syrian Pound"                                                     
#> [149] "Taka"                                                             
#> [150] "Tala"                                                             
#> [151] "Tanzanian Shilling"                                               
#> [152] "Tenge"                                                            
#> [153] "The codes assigned for transactions where no currency is involved"
#> [154] "Trinidad and Tobago Dollar"                                       
#> [155] "Tugrik"                                                           
#> [156] "Tunisian Dinar"                                                   
#> [157] "Turkish Lira"                                                     
#> [158] "Turkmenistan New Manat"                                           
#> [159] "UAE Dirham"                                                       
#> [160] "Uganda Shilling"                                                  
#> [161] "Unidad de Fomento"                                                
#> [162] "Unidad de Valor Real"                                             
#> [163] "Unidad Previsional"                                               
#> [164] "Uruguay Peso en Unidades Indexadas (UI)"                          
#> [165] "US Dollar"                                                        
#> [166] "US Dollar (Next day)"                                             
#> [167] "Uzbekistan Sum"                                                   
#> [168] "Vatu"                                                             
#> [169] "WIR Euro"                                                         
#> [170] "WIR Franc"                                                        
#> [171] "Won"                                                              
#> [172] "Yemeni Rial"                                                      
#> [173] "Yen"                                                              
#> [174] "Yuan Renminbi"                                                    
#> [175] "Zambian Kwacha"                                                   
#> [176] "Zimbabwe Dollar"                                                  
#> [177] "Zloty"

Standardise variable values

We standardise the target variable values, specifying that we are using the lookup codes method and not the fuzzy-logic / correspondences method.

X <- ratify(X, type_1L_chr = "Lookup")

This significantly reduces the umber of non-standard values for our target variable.

X@results_ls$Currency_Output_Validation$Invalid_Values
#>  [1] "Bitcoin[4] (as legal tender)" "Cook Islands dollar"          "Faroese króna"                "Guernsey pound"              
#>  [5] "Jersey pound"                 "Kiribati dollar[E]"           "Manx pound"                   "Niue dollar[E]"              
#>  [9] "Pitcairn Islands dollar[E]"   "Sahrawi peseta"               "Tuvaluan dollar"              "Zimbabwean dollar"

If we wish we can remove the non-standardised values.

X <- renew(X, T, type_1L_chr = "slot", what_1L_chr = "force") 
X <- ratify(X, type_1L_chr = "Lookup")

We can no inspect our results a dataset for which the country names and currency names now conform to ISO standards.

X@results_ls$Currency_Output_Lookup %>%
  renew(type_1L_chr = "label") %>%
  exhibit(scroll_box_args_ls = list(width = "100%"))

Dataset
Country name	Currency name	Currency symbol	Currency alphabetical ISO code (three letter)	Currency's fractional unit	Number of fractional units in basic unit
Afghanistan	Afghani	؋‎	AFN	Pul	100
Albania	Lek	Lek	ALL	Qintar	100
Algeria	Algerian Dinar	DA	DZD	Santeem	100
Andorra	Euro	€	EUR	Cent	100
Angola	Kwanza	Kz	AOA	Cêntimo	100
Anguilla	East Caribbean Dollar	\$	XCD	Cent	100
Antigua and Barbuda	East Caribbean Dollar	\$	XCD	Cent	100
Argentina	Argentine Peso	\$	ARS	Centavo	100
Armenia	Armenian Dram	֏	AMD	Luma	100
Aruba	Aruban Florin	ƒ	AWG	Cent	100
Saint Helena, Ascension and Tristan da Cunha	Saint Helena Pound	£	SHP	Penny	100
Australia	Australian Dollar	\$	AUD	Cent	100
Austria	Euro	€	EUR	Cent	100
Azerbaijan	Azerbaijan Manat	₼	AZN	Qəpik	100
Bahamas	Bahamian Dollar	\$	BSD	Cent	100
Bahrain	Bahraini Dinar	BD	BHD	Fils	1000
Bangladesh	Taka	৳	BDT	Poisha	100
Barbados	Barbados Dollar	\$	BBD	Cent	100
Belarus	Belarusian Ruble	Br	BYN	Copeck	100
Belgium	Euro	€	EUR	Cent	100
Belize	Belize Dollar	\$	BZD	Cent	100
Benin	CFA Franc BCEAO	Fr	XOF	Centime	100
Bermuda	Bermudian Dollar	\$	BMD	Cent	100
Bhutan	Ngultrum	Nu	BTN	Chetrum	100
Bhutan	Indian Rupee	₹	INR	Paisa	100
Bolivia, Plurinational State of	Boliviano	Bs	BOB	Centavo	100
Bonaire, Sint Eustatius and Saba	US Dollar	\$	USD	Cent	100
Bosnia and Herzegovina	Convertible Mark	KM	BAM	Fening	100
Botswana	Pula	P	BWP	Thebe	100
Brazil	Brazilian Real	R\$	BRL	Centavo	100
British Indian Ocean Territory	US Dollar	\$	USD	Cent	100
Virgin Islands, British	US Dollar	\$	USD	Cent	100
Brunei Darussalam	Brunei Dollar	\$	BND	Sen	10
Brunei Darussalam	Singapore Dollar	\$	SGD	Cent	10
Bulgaria	Bulgarian Lev	Lev	BGN	Stotinka	100
Burkina Faso	CFA Franc BCEAO	Fr	XOF	Centime	100
Burundi	Burundi Franc	Fr	BIF	Centime	100
Cambodia	Riel	៛	KHR	Sen	100
Cambodia	US Dollar	\$	USD	Cent	100
Cameroon	CFA Franc BEAC	Fr	XAF	Centime	100
Canada	Canadian Dollar	\$	CAD	Cent	100
Cabo Verde	Cabo Verde Escudo	\$	CVE	Centavo	100
Cayman Islands	Cayman Islands Dollar	\$	KYD	Cent	100
Central African Republic	CFA Franc BEAC	Fr	XAF	Centime	100
Chad	CFA Franc BEAC	Fr	XAF	Centime	100
Chile	Chilean Peso	\$	CLP	Centavo	100
China	Yuan Renminbi	¥	CNY	Jiao\[G\]	10
Colombia	Colombian Peso	\$	COP	Centavo	100
Comoros	Comorian Franc	Fr	KMF	Centime	100
Congo, The Democratic Republic of the	Congolese Franc	Fr	CDF	Centime	100
Congo	CFA Franc BEAC	Fr	XAF	Centime	100
Cook Islands	New Zealand Dollar	\$	NZD	Cent	100
Costa Rica	Costa Rican Colon	₡	CRC	Céntimo	100
Côte d'Ivoire	CFA Franc BCEAO	Fr	XOF	Centime	100
Croatia	Euro	€	EUR	Cent	100
Cuba	Cuban Peso	\$	CUP	Centavo	100
Curaçao	Netherlands Antillean Guilder	ƒ	ANG	Cent	100
Cyprus	Euro	€	EUR	Cent	100
Czechia	Czech Koruna	Kč	CZK	Heller	100
Denmark	Danish Krone	kr	DKK	Øre	100
Djibouti	Djibouti Franc	Fr	DJF	Centime	100
Dominica	East Caribbean Dollar	\$	XCD	Cent	100
Dominican Republic	Dominican Peso	\$	DOP	Centavo	100
Timor-Leste	US Dollar	\$	USD	Centavo	100
Ecuador	US Dollar	\$	USD	Centavo	100
Egypt	Egyptian Pound	LE	EGP	Piastre\[B\]	100
El Salvador	US Dollar	\$	USD	Cent	100
Equatorial Guinea	CFA Franc BEAC	Fr	XAF	Centime	100
Eritrea	Nakfa	Nkf	ERN	Cent	100
Estonia	Euro	€	EUR	Cent	100
Eswatini	Lilangeni	L or E (pl.)	SZL	Cent	100
Eswatini	Rand	R	ZAR	Cent	100
Ethiopia	Ethiopian Birr	Br	ETB	Santim	100
Falkland Islands (Malvinas)	Falkland Islands Pound	£	FKP	Penny	100
Falkland Islands (Malvinas)	Pound Sterling	£	GBP	Penny	100
Faroe Islands	Danish Krone	kr	DKK	Øre	100
Fiji	Fiji Dollar	\$	FJD	Cent	100
Finland	Euro	€	EUR	Cent	100
France	Euro	€	EUR	Cent	100
French Polynesia	CFP Franc	Fr	XPF	Centime	100
French Southern Territories	Euro	€	EUR	Cent	100
Gabon	CFA Franc BEAC	Fr	XAF	Centime	100
Gambia	Dalasi	D	GMD	Butut	100
Georgia	Lari	₾	GEL	Tetri	100
Germany	Euro	€	EUR	Cent	100
Ghana	Ghana Cedi	₵	GHS	Pesewa	100
Gibraltar	Gibraltar Pound	£	GIP	Penny	100
Gibraltar	Pound Sterling	£	GBP	Penny	100
Greece	Euro	€	EUR	Cent	100
Greenland	Danish Krone	kr	DKK	Øre	100
Grenada	East Caribbean Dollar	\$	XCD	Cent	100
Guatemala	Quetzal	Q	GTQ	Centavo	100
Guernsey	Pound Sterling	£	GBP	Penny	100
Guinea	Guinean Franc	Fr	GNF	Centime	100
Guinea-Bissau	CFA Franc BCEAO	Fr	XOF	Centime	100
Guyana	Guyana Dollar	\$	GYD	Cent	100
Haiti	Gourde	G	HTG	Centime	100
Honduras	Lempira	L	HNL	Centavo	100
Hong Kong	Hong Kong Dollar	\$	HKD	Cent	100
Hungary	Forint	Ft	HUF	Fillér	100
Iceland	Iceland Krona	kr	ISK	Eyrir	100
India	Indian Rupee	₹	INR	Paisa	100
Indonesia	Rupiah	Rp	IDR	Sen	100
Iran, Islamic Republic of	Iranian Rial	Rl or Rls (pl.)	IRR	Rial	1
Iraq	Iraqi Dinar	ID	IQD	Fils	1000
Ireland	Euro	€	EUR	Cent	100
Isle of Man	Pound Sterling	£	GBP	Penny	100
Israel	New Israeli Sheqel	₪	ILS	Agora	100
Italy	Euro	€	EUR	Cent	100
Jamaica	Jamaican Dollar	\$	JMD	Cent	100
Japan	Yen	¥	JPY	Sen\[C\]	100
Jersey	Pound Sterling	£	GBP	Penny	100
Jordan	Jordanian Dinar	JD	JOD	Piastre\[H\]	100
Kazakhstan	Tenge	₸	KZT	Tıyn	100
Kenya	Kenyan Shilling	Sh or Shs (pl.)	KES	Cent	100
Kiribati	Australian Dollar	\$	AUD	Cent	100
Korea, Democratic People's Republic of	North Korean Won	₩	KPW	Chon	100
Korea, Republic of	Won	₩	KRW	Jeon	100
Kuwait	Kuwaiti Dinar	KD	KWD	Fils	1000
Kyrgyzstan	Som	som	KGS	Tyiyn	100
Lao People's Democratic Republic	Lao Kip	₭	LAK	Att	100
Latvia	Euro	€	EUR	Cent	100
Lebanon	Lebanese Pound	LL	LBP	Piastre	100
Lesotho	Loti	L or M (pl.)	LSL	Sente	100
Lesotho	Rand	R	ZAR	Cent	100
South Georgia and the South Sandwich Islands	Falkland Islands Pound	£	FKP	Penny	100
South Georgia and the South Sandwich Islands	Pound Sterling	£	GBP	Penny	100
Liberia	Liberian Dollar	\$	LRD	Cent	100
Liberia	US Dollar	\$	USD	Cent	100
Libya	Libyan Dinar	LD	LYD	Dirham	1000
Liechtenstein	Swiss Franc	Fr	CHF	Rappen	100
Lithuania	Euro	€	EUR	Cent	100
Luxembourg	Euro	€	EUR	Cent	100
Macao	Pataca	MOP\$	MOP	Avo	100
Macao	Hong Kong Dollar	\$	HKD	Cent	100
Madagascar	Malagasy Ariary	Ar	MGA	Iraimbilanja	5
Malawi	Malawi Kwacha	K	MWK	Tambala	100
Malaysia	Malaysian Ringgit	RM	MYR	Sen	100
Maldives	Rufiyaa	Rf	MVR	Laari	100
Mali	CFA Franc BCEAO	Fr	XOF	Centime	100
Malta	Euro	€	EUR	Cent	100
Marshall Islands	US Dollar	\$	USD	Cent	100
Mauritania	Ouguiya	UM	MRU	Khoums	5
Mauritius	Mauritius Rupee	Re or Rs (pl.)	MUR	Cent	100
Mexico	Mexican Peso	\$	MXN	Centavo	100
Micronesia, Federated States of	US Dollar	\$	USD	Cent	100
Moldova, Republic of	Moldovan Leu	Leu or Lei (pl.)	MDL	Ban	100
Monaco	Euro	€	EUR	Cent	100
Mongolia	Tugrik	₮	MNT	Möngö	100
Montenegro	Euro	€	EUR	Cent	100
Montserrat	East Caribbean Dollar	\$	XCD	Cent	100
Morocco	Moroccan Dirham	DH	MAD	Centime	100
Mozambique	Mozambique Metical	Mt	MZN	Centavo	100
Myanmar	Kyat	K or Ks (pl.)	MMK	Pya	100
Namibia	Namibia Dollar	\$	NAD	Cent	100
Namibia	Rand	R	ZAR	Cent	100
Nauru	Australian Dollar	\$	AUD	Cent	100
Nepal	Nepalese Rupee	Re or Rs (pl.)	NPR	Paisa	100
Nepal	Indian Rupee	₹	INR	Paisa	100
Netherlands	Euro	€	EUR	Cent	100
New Caledonia	CFP Franc	Fr	XPF	Centime	100
New Zealand	New Zealand Dollar	\$	NZD	Cent	100
Nicaragua	Cordoba Oro	C\$	NIO	Centavo	100
Niger	CFA Franc BCEAO	Fr	XOF	Centime	100
Nigeria	Naira	₦	NGN	Kobo	100
Niue	New Zealand Dollar	\$	NZD	Cent	100
North Macedonia	Denar	DEN	MKD	Deni	100
Norway	Norwegian Krone	kr	NOK	Øre	100
Oman	Rial Omani	RO	OMR	Baisa	1000
Pakistan	Pakistan Rupee	Re or Rs (pl.)	PKR	Paisa	100
Palau	US Dollar	\$	USD	Cent	100
Palestine, State of	New Israeli Sheqel	₪	ILS	Agora	100
Palestine, State of	Jordanian Dinar	JD	JOD	Piastre\[H\]	100
Panama	Balboa	B/	PAB	Centésimo	100
Panama	US Dollar	\$	USD	Cent	100
Papua New Guinea	Kina	K	PGK	Toea	100
Paraguay	Guarani	₲	PYG	Céntimo	100
Peru	Sol	S/	PEN	Céntimo	100
Philippines	Philippine Peso	₱	PHP	Sentimo	100
Pitcairn	New Zealand Dollar	\$	NZD	Cent	100
Poland	Zloty	zł	PLN	Grosz	100
Portugal	Euro	€	EUR	Cent	100
Qatar	Qatari Rial	QR	QAR	Dirham	100
Romania	Romanian Leu	Leu or Lei (pl.)	RON	Ban	100
Russian Federation	Russian Ruble	₽	RUB	Kopeck	100
Rwanda	Rwanda Franc	Fr	RWF	Centime	100
Bonaire, Sint Eustatius and Saba	US Dollar	\$	USD	Cent	100
Western Sahara	Moroccan Dirham	DH	MAD	Centime	100
Saint Helena, Ascension and Tristan da Cunha	Saint Helena Pound	£	SHP	Penny	100
Saint Helena, Ascension and Tristan da Cunha	Pound Sterling	£	GBP	Penny	100
Saint Kitts and Nevis	East Caribbean Dollar	\$	XCD	Cent	100
Saint Lucia	East Caribbean Dollar	\$	XCD	Cent	100
Saint Pierre and Miquelon	Euro	€	EUR	Cent	100
Saint Pierre and Miquelon	Canadian Dollar	\$	CAD	Cent	100
Samoa	Tala	\$	WST	Sene	100
Saint Barthélemy	Euro	€	EUR	Cent	100
San Marino	Euro	€	EUR	Cent	100
Sao Tome and Principe	Dobra	Db	STN	Cêntimo	100
Saudi Arabia	Saudi Riyal	Rl or Rls (pl.)	SAR	Halala	100
Senegal	CFA Franc BCEAO	Fr	XOF	Centime	100
Serbia	Serbian Dinar	DIN	RSD	Para	100
Seychelles	Seychelles Rupee	Re or Rs (pl.)	SCR	Cent	100
Sierra Leone	Leone	Le	SLE	Cent	100
Singapore	Singapore Dollar	\$	SGD	Cent	10
Singapore	Brunei Dollar	\$	BND	Sen	10
Bonaire, Sint Eustatius and Saba	US Dollar	\$	USD	Cent	100
Sint Maarten (Dutch part)	Netherlands Antillean Guilder	ƒ	ANG	Cent	100
Slovakia	Euro	€	EUR	Cent	100
Slovenia	Euro	€	EUR	Cent	100
Solomon Islands	Solomon Islands Dollar	\$	SBD	Cent	100
Somalia	Somali Shilling	Sh or Shs (pl.)	SOS	Cent	100
South Africa	Rand	R	ZAR	Cent	100
South Sudan	South Sudanese Pound	(none)	SSP	Piaster	100
Spain	Euro	€	EUR	Cent	100
Sri Lanka	Sri Lanka Rupee	Re or Rs (pl.)	LKR	Cent	100
Sudan	Sudanese Pound	LS	SDG	Piastre	100
Suriname	Surinam Dollar	\$	SRD	Cent	100
Sweden	Swedish Krona	kr	SEK	Öre	100
Switzerland	Swiss Franc	Fr	CHF	Rappen\[J\]	100
Syrian Arab Republic	Syrian Pound	LS	SYP	Piastre	100
Taiwan, Province of China	New Taiwan Dollar	\$	TWD	Cent	100
Tajikistan	Somoni	SM	TJS	Diram	100
Tanzania, United Republic of	Tanzanian Shilling	Sh or Shs (pl.)	TZS	Cent	100
Thailand	Baht	฿	THB	Satang	100
Togo	CFA Franc BCEAO	Fr	XOF	Centime	100
Tonga	Pa'anga	T\$	TOP	Seniti	100
Trinidad and Tobago	Trinidad and Tobago Dollar	\$	TTD	Cent	100
Tunisia	Tunisian Dinar	DT	TND	Millime	1000
Turkey	Turkish Lira	₺	TRY	Kuruş	100
Turkmenistan	Turkmenistan New Manat	m	TMT	Tenge	100
Turks and Caicos Islands	US Dollar	\$	USD	Cent	100
Tuvalu	Australian Dollar	\$	AUD	Cent	100
Uganda	Uganda Shilling	Sh or Shs (pl.)	UGX	(none)	(none)
Ukraine	Hryvnia	₴	UAH	Kopeck	100
United Arab Emirates	UAE Dirham	Dh or Dhs (pl.)	AED	Fils	100
United Kingdom	Pound Sterling	£	GBP	Penny	100
United States	US Dollar	\$	USD	Cent\[A\]	100
Uruguay	Peso Uruguayo	\$	UYU	Centésimo	100
Uzbekistan	Uzbekistan Sum	soum	UZS	Tiyin	100
Vanuatu	Vatu	VT	VUV	Cent	100
Holy See (Vatican City State)	Euro	€	EUR	Cent	100
Venezuela, Bolivarian Republic of	Bolívar Soberano	Bs.S	VES	Céntimo	1
Venezuela, Bolivarian Republic of	Bolívar Soberano	Bs.D	VED	Céntimo	100
Viet Nam	Dong	₫	VND	Hào\[L\]	10
Wallis and Futuna	CFP Franc	Fr	XPF	Centime	100
Yemen	Yemeni Rial	Rl or Rls (pl.)	YER	Fils	100
Zambia	Zambian Kwacha	K	ZMW	Ngwee	100
Zimbabwe	US Dollar	\$	USD	Cent	100

Docs: Add metadata to datasets of individual human records

Fri, 20 Oct 2023 00:00:00 +0000

This below section renders a vignette article from the youthvars library. You can use the following links to:

view the vignette on the library website (adds useful hyperlinks to code blocks)
view the source file from that article, and;
edit its contents (requires a GitHub account).

Note: This vignette is illustrated with fake data. The dataset explored in this example should not be used to inform decision-making.

library(ready4)
library(youthvars)

Youthvars provides two ready4 framework modules - YouthvarsProfile and YouthvarsSeries that form part of the ready4 economic model of youth mental health. The ready4 modules in youthvars extend the Ready4useDyad module and can be used to help describe key structural properties of youth mental health datasets.

Ingest data

To start we ingest X, a Ready4useDyad (dataset and data dictionary pair) that we can download from a remote repository.

X <- ready4use::Ready4useRepos(dv_nm_1L_chr = "fakes",
                               dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/W95KED",
                               dv_server_1L_chr = "dataverse.harvard.edu") %>%
  ingest(fls_to_ingest_chr = "ymh_clinical_dyad_r4",
         metadata_1L_lgl = F)

Add metadata

If a dataset is cross-sectional or we wish to treat it as if it were (i.e., where data collection rounds are ignored) we can create Y, an instance of the YouthvarsProfile module, to add minimal metadata (the name of the unique identifier variable).

Y <- YouthvarsProfile(a_Ready4useDyad = X, id_var_nm_1L_chr = "fkClientID")

If the temporal dimension of the dataset is important, it may be therefore preferable to instead transform X into a YouthvarsSeries module instance. YouthvarsSeries objects contain all of the fields of YouthvarsProfile objects, but also include additional fields that are specific for longitudinal datasets (e.g. timepoint_var_nm_1L_chr and timepoint_vals_chr that respectively specify the data-collection timepoint variable name and values and participation_var_1L_chr that specifies the desired name of a yet to be created variable that will summarise the data-collection timepoints for which each unit record supplied data).

Z <- YouthvarsSeries(a_Ready4useDyad = X,
                     id_var_nm_1L_chr = "fkClientID",
                     participation_var_1L_chr = "participation",
                     timepoint_vals_chr = c("Baseline","Follow-up"),
                     timepoint_var_nm_1L_chr = "round")

YouthvarsProfile methods

Inspect data

We can now specify the variables that we would like to prepare descriptive statistics for by using the renew method. The variables to be profiled are specified in the profile_chr argument, the number of decimal digits (default = 3) of numeric values in the summary tables to be generated can be specified with nbr_of_digits_1L_int.

Y <- renew(Y, nbr_of_digits_1L_int = 2L, profile_chr = c("d_age","d_sexual_ori_s","d_studying_working"))

We can now view the descriptive statistics we created in the previous step.

Y %>%
  exhibit(profile_idx_int = 1L, scroll_box_args_ls = list(width = "100%"))

We can also plot the distributions of selected variables in our dataset.

depict(Y, var_nms_chr = c("c_sofas"), labels_chr = c("SOFAS"))

SOFAS total scores

YouthvarsSeries methods

Validate data

To explore longitudinal data we need to first use the ratify method to ensure that Z has been appropriately configured for methods examining datasets reporting measures at two timepoints.

Z <- ratify(Z,
            type_1L_chr = "two_timepoints")

Inspect data

We can now specify the variables that we would like to prepare descriptive statistics for using the renew method. The variables to be profiled are specified in arguments beginning with “compare_”. Use compare_ptcpn_chr to compare variables based on whether cases reported data at one or both timepoints and compare_by_time_chr to compare the summary statistics of variables by timepoints, e.g at baseline and follow-up. If you wish these comparisons to report p values, then use the compare_ptcpn_with_test_chr and compare_by_time_with_test_chr arguments.

Z <- renew(Z,
           compare_by_time_chr = c("d_age","d_sexual_ori_s","d_studying_working"),
           compare_by_time_with_test_chr = c("k6_total", "phq9_total", "bads_total"),
           compare_ptcpn_with_test_chr = c("k6_total", "phq9_total", "bads_total"))

The tables generated in the preceding step can be inspected using the exhibit method.

Z %>%
  exhibit(profile_idx_int = 1L,
          scroll_box_args_ls = list(width = "100%"))

Z %>%
  exhibit(profile_idx_int = 2L,
          scroll_box_args_ls = list(width = "100%"))

Z %>%
  exhibit(profile_idx_int = 3L,
          scroll_box_args_ls = list(width = "100%"))

The depict method can create plots, comparing numeric variables by timepoint.

depict(Z,
       type_1L_chr = "by_time",
       var_nms_chr = c("c_sofas"),
       label_fill_1L_chr = "Time",#
       labels_chr = c("SOFAS"),#
       y_label_1L_chr = "")

SOFAS total scores by data collection round

If and only if the dataset you are working with is appropriate for public dissemination (e.g. is synthetic data), you can use the following workflow for sharing it. We can share the dataset we created for this example using the share method, specifying the repository to which we wish to publish the dataset (and for which we have write permissions) in a (Ready4useRepos object).

A <- Ready4useRepos(gh_repo_1L_chr = "ready4-dev/youthvars", # Replace with your repository 
                          gh_tag_1L_chr = "Documentation_0.0"), # (need write permissions).
A <- share(A,
           obj_to_share_xx = Z,
           fl_nm_1L_chr = "ymh_YouthvarsSeries")

Z is now available for download as the file ymh_YouthvarsSeries.RDS from the “Documentation_0.0” release of the youthvars package.

Docs: Validate variable total scores

Fri, 20 Oct 2023 00:00:00 +0000

This below section renders a vignette article from the youthvars library. You can use the following links to:

view the vignette on the library website (adds useful hyperlinks to code blocks)
view the source file from that article, and;
edit its contents (requires a GitHub account).

library(ready4)
library(youthvars)

Variable classes and data integrity

The youthvars package includes a number of ready4 framework sub-module classes that form part of the ready4 economic model of youth mental health. The primary use of youthvars sub-modules is to quality assure the variables used in model input and output datasets by:

facilitating automated data integrity checks that verify no impermissible values (e.g. utility scores greater than one) are present in source data, transformed data or results; and
implementing rules-based automated selection and application of appropriate methods for each dataset variable.

Included sub-module classes

The initial set of sub-module classes included in the youthvars package are one class for Assessment of Quality of Life (Adolescent) health utility and one for each of the predictors used in the utility prediction algorithms included in the related youthu package.

Assessment of Quality of Life Six Dimension (Adolescent) Health Utility

The youthvars_aqol6d_adol class is defined for numeric vectors with a minimum value of 0.03 and maximum value of 1.0.

youthvars_aqol6d_adol(0.4)
#> [1] 0.4
#> attr(,"class")
#> [1] "youthvars_aqol6d_adol" "numeric"

youthvars_aqol6d_adol(c(0.03,0.2,1))
#> [1] 0.03 0.20 1.00
#> attr(,"class")
#> [1] "youthvars_aqol6d_adol" "numeric"

Non numeric objects and values outside these ranges will produce errors.

youthvars_aqol6d_adol("0.5")
#> Error in make_new_youthvars_aqol6d_adol(x): is.numeric(x) is not TRUE

youthvars_aqol6d_adol(-0.1)
#> Error: All non-missing values in valid youthvars_aqol6d_adol object must be greater than or equal to 0.03.

youthvars_aqol6d_adol(1.2)
#> Error: All non-missing values in valid youthvars_aqol6d_adol object must be less than or equal to 1.

Child Health Utility Nine Dimension - Australian Adolescent Scoring

The youthvars_chu9d_adolaus class is defined for numeric vectors with a minimum value of -0.2118 and maximum value of 1.0.

youthvars_chu9d_adolaus(0.4)
#> [1] 0.4
#> attr(,"class")
#> [1] "youthvars_chu9d_adolaus" "numeric"

youthvars_chu9d_adolaus(c(0.03,0.2,1))
#> [1] 0.03 0.20 1.00
#> attr(,"class")
#> [1] "youthvars_chu9d_adolaus" "numeric"

Non numeric objects and values outside these ranges will produce errors.

youthvars_chu9d_adolaus("0.5")
#> Error in make_new_youthvars_chu9d_adolaus(x): is.numeric(x) is not TRUE

youthvars_chu9d_adolaus(-0.3)
#> Error: All non-missing values in valid youthvars_chu9d_adolaus object must be greater than or equal to -0.2118.

youthvars_chu9d_adolaus(1.2)
#> Error: All non-missing values in valid youthvars_chu9d_adolaus object must be less than or equal to 1.

Behavioural Activation for Depression Scale (BADS)

The youthvars_bads class is defined for integer vectors with a minimum value of 0 and maximum value of 150.

youthvars_bads(143L)
#> [1] 143
#> attr(,"class")
#> [1] "youthvars_bads" "integer"

youthvars_bads(as.integer(c(1,15,150)))
#> [1]   1  15 150
#> attr(,"class")
#> [1] "youthvars_bads" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_bads(22.5)
#> Error in make_new_youthvars_bads(x): is.integer(x) is not TRUE

youthvars_bads(-1L)
#> Error: All non-missing values in valid youthvars_bads object must be greater than or equal to 0.

youthvars_bads(160L)
#> Error: All non-missing values in valid youthvars_bads object must be less than or equal to 150.

Generalised Anxiety Disorder Scale (GAD-7)

The youthvars_gad7 class is defined for integer vectors with a minimum value of 0 and a maximum value of 21.

youthvars_gad7(15L)
#> [1] 15
#> attr(,"class")
#> [1] "youthvars_gad7" "integer"

youthvars_gad7(as.integer(c(0,14,21)))
#> [1]  0 14 21
#> attr(,"class")
#> [1] "youthvars_gad7" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_gad7(14.6)
#> Error in make_new_youthvars_gad7(x): is.integer(x) is not TRUE

youthvars_gad7(-1L)
#> Error: All non-missing values in valid youthvars_gad7 object must be greater than or equal to 0.

youthvars_gad7(22L)
#> Error: All non-missing values in valid youthvars_gad7 object must be less than or equal to 21.

Kessler Psychological Distress Scale (K6) - Australian Scoring System

The youthvars_k6_aus class is defined for integer vectors with a minimum value of 6 and a maximum value of 30.

youthvars_k6_aus(21L)
#> [1] 21
#> attr(,"class")
#> [1] "youthvars_k6_aus" "integer"

youthvars_k6_aus(as.integer(c(6,13,25)))
#> [1]  6 13 25
#> attr(,"class")
#> [1] "youthvars_k6_aus" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_k6_aus(11.2)
#> Error in make_new_youthvars_k6_aus(x): is.integer(x) is not TRUE

youthvars_k6_aus(1L)
#> Error: All non-missing values in valid youthvars_k6_aus object must be greater than or equal to 6.

youthvars_k6_aus(31L)
#> Error: All non-missing values in valid youthvars_k6_aus object must be less than or equal to 30.

Kessler Psychological Distress Scale (K6) - US Scoring System

The youthvars_k6 class is defined for integer vectors with a minimum value of 0 and a maximum value of 24.

youthvars_k6(21L)
#> [1] 21
#> attr(,"class")
#> [1] "youthvars_k6" "integer"

youthvars_k6(as.integer(c(0,13,24)))
#> [1]  0 13 24
#> attr(,"class")
#> [1] "youthvars_k6" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_k6(11.2)
#> Error in make_new_youthvars_k6(x): is.integer(x) is not TRUE

youthvars_k6(-1L)
#> Error: All non-missing values in valid youthvars_k6 object must be greater than or equal to 0.

youthvars_k6(25L)
#> Error: All non-missing values in valid youthvars_k6 object must be less than or equal to 24.

Kessler Psychological Distress Scale (K10) - Australian Scoring System

The youthvars_k10_aus class is defined for integer vectors with a minimum value of 10 and a maximum value of 50.

youthvars_k10_aus(21L)
#> [1] 21
#> attr(,"class")
#> [1] "youthvars_k10_aus" "integer"

youthvars_k10_aus(as.integer(c(13,25,41)))
#> [1] 13 25 41
#> attr(,"class")
#> [1] "youthvars_k10_aus" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_k10_aus(11.2)
#> Error in make_new_youthvars_k10_aus(x): is.integer(x) is not TRUE

youthvars_k10_aus(9L)
#> Error: All non-missing values in valid youthvars_k10_aus object must be greater than or equal to 10.

youthvars_k10_aus(51L)
#> Error: All non-missing values in valid youthvars_k10_aus object must be less than or equal to 50.

Kessler Psychological Distress Scale (K10) - US Scoring System

The youthvars_k10 class is defined for integer vectors with a minimum value of 0 and a maximum value of 40.

youthvars_k10(21L)
#> [1] 21
#> attr(,"class")
#> [1] "youthvars_k10" "integer"

youthvars_k10(as.integer(c(0,13,34)))
#> [1]  0 13 34
#> attr(,"class")
#> [1] "youthvars_k10" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_k10(11.2)
#> Error in make_new_youthvars_k10(x): is.integer(x) is not TRUE

youthvars_k10(-1L)
#> Error: All non-missing values in valid youthvars_k10 object must be greater than or equal to 0.

youthvars_k10(41L)
#> Error: All non-missing values in valid youthvars_k10 object must be less than or equal to 40.

Overall Anxiety Severity and Impairment Scale (OASIS)

The youthvars_oasis class is defined for integer vectors with a minimum value of 0 and a maximum value of 20.

youthvars_oasis(15L)
#> [1] 15
#> attr(,"class")
#> [1] "youthvars_oasis" "integer"

youthvars_oasis(as.integer(c(0,12,20)))
#> [1]  0 12 20
#> attr(,"class")
#> [1] "youthvars_oasis" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_oasis(14.2)
#> Error in make_new_youthvars_oasis(x): is.integer(x) is not TRUE

youthvars_oasis(-1L)
#> Error: All non-missing values in valid youthvars_oasis object must be greater than or equal to 0.

youthvars_oasis(21L)
#> Error: All non-missing values in valid youthvars_oasis object must be less than or equal to 20.

Patient Health Questionnaire (PHQ-9)

The youthvars_phq9 class is defined for integer vectors with a minimum value of 0 and a maximum value of 27.

youthvars_phq9(11L)
#> [1] 11
#> attr(,"class")
#> [1] "youthvars_phq9" "integer"

youthvars_phq9(as.integer(c(0,13,27)))
#> [1]  0 13 27
#> attr(,"class")
#> [1] "youthvars_phq9" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_phq9(15.2)
#> Error in make_new_youthvars_phq9(x): is.integer(x) is not TRUE

youthvars_phq9(-1L)
#> Error: All non-missing values in valid youthvars_phq9 object must be greater than or equal to 0.

youthvars_phq9(28L)
#> Error: All non-missing values in valid youthvars_phq9 object must be less than or equal to 27.

The youthvars_scared class is defined for integer vectors with a minimum value of 0 and a maximum value of 82.

youthvars_scared(77L)
#> [1] 77
#> attr(,"class")
#> [1] "youthvars_scared" "integer"

youthvars_scared(as.integer(c(0,42,82)))
#> [1]  0 42 82
#> attr(,"class")
#> [1] "youthvars_scared" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_scared(33.2)
#> Error in make_new_youthvars_scared(x): is.integer(x) is not TRUE

youthvars_scared(-1L)
#> Error: All non-missing values in valid youthvars_scared object must be greater than or equal to 0.

youthvars_scared(83)
#> Error in make_new_youthvars_scared(x): is.integer(x) is not TRUE

The youthvars_sofas class is defined for integer vectors with a minimum value of 0 and a maximum value of 100.

youthvars_sofas(44L)
#> [1] 44
#> attr(,"class")
#> [1] "youthvars_sofas" "integer"

youthvars_sofas(as.integer(c(0,23,89)))
#> [1]  0 23 89
#> attr(,"class")
#> [1] "youthvars_sofas" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_sofas(73.2)
#> Error in make_new_youthvars_sofas(x): is.integer(x) is not TRUE

youthvars_sofas(-1L)
#> Error: All non-missing values in valid youthvars_sofas object must be greater than or equal to 0.

youthvars_sofas(103L)
#> Error: All non-missing values in valid youthvars_sofas object must be less than or equal to 100.

Docs: Ingest, label and share model data

Thu, 05 Jan 2017 00:00:00 +0000

Docs: Share data via online repositories

Wed, 28 Jan 2026 00:00:00 +0000

This below section renders a vignette article from the ready4use library. You can use the following links to:

view the vignette on the library website (adds useful hyperlinks to code blocks)
view the source file from that article, and;
edit its contents (requires a GitHub account).

Note: This vignette is illustrated with fake data. The dataset explored in this example should not be used to inform decision-making.

library(ready4)
library(ready4use)

ready4use includes a number of tools for sharing health economic model data that forms part of the ready4 framework.

Identify data to be shared

To illustrate how to share data using ready4use classes and methods, we will first need some data to publish. In this example, we are going to share X, a Ready4useDyad (a data structure explained in another vignette) that we can create using data ingested from an online repository.

objects_ls <- ingest(Ready4useRepos(dv_nm_1L_chr = "fakes",
                                    dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/HJXYKQ",
                                    dv_server_1L_chr = "dataverse.harvard.edu",
                                    gh_repo_1L_chr = "ready4-dev/ready4",
                                    gh_tag_1L_chr = "Documentation_0.0"),
                     fls_to_ingest_chr = c("ymh_clinical_tb","ymh_clinical_dict_r3"),
                     metadata_1L_lgl = F)
X <- Ready4useDyad(ds_tb = objects_ls$ymh_clinical_tb,
                   dictionary_r3 = objects_ls$ymh_clinical_dict_r3) %>%
  renew()

We now specify where we plan to publish X in Y, a Ready4useRepos object (described in another vignette). Note, you must have write permissions to the repositories you specify in this step. The values entered in this example (the https://doi.org/10.7910/DVN/W95KED dataset from the fakes dataverse will not work for you).

Y <- Ready4useRepos(dv_nm_1L_chr = "fakes", # Replace with values for a dataverse & dataset for which
                    dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/W95KED", #  you have write permissions.
                    dv_server_1L_chr = "dataverse.harvard.edu")

We can now upload X to our preferred data repository using the share method. By default, if more than one data repository was specified in Y, then the dataverse repository will be preferred when sharing. We can overwrite this default by passing either “prefer_gh” or “all” values to the type_1L_chr argument. The Ready4useDyad object is now available for download at https://doi.org/10.7910/DVN/W95KED.

Y <- share(Y,
           obj_to_share_xx = X,
           fl_nm_1L_chr = "ymh_clinical_dyad_r4",
           description_1L_chr = "An example of a Ready4useDyad - a dataset (clinical youth mental health, AQoL-6D) and data dictionary pair. Note this example uses fake data.")

ready4 – Data - management

Docs: Standardise Variable Values With Fuzzy Logic And Correspondence Tables

In brief

Create project

Supply seed dataset

Specify standards

Compare variable of interest values from seed and standards dataset.

Standardise variable values

Docs: Add a data dictionary to a dataset

Create a dataset-dictionary pair

Inspect data

Docs: Standardise Variable Values With Lookup Codes

In brief

Create project

Supply seed dataset

Specify standards

Compare variable of interest values from seed and standards dataset.

Standardise variable values

Docs: Add metadata to datasets of individual human records

Ingest data

Add metadata

YouthvarsProfile methods

Inspect data

YouthvarsSeries methods

Validate data

Inspect data

Share data

Docs: Validate variable total scores

Variable classes and data integrity

Included sub-module classes

Assessment of Quality of Life Six Dimension (Adolescent) Health Utility

Child Health Utility Nine Dimension - Australian Adolescent Scoring

Behavioural Activation for Depression Scale (BADS)

Generalised Anxiety Disorder Scale (GAD-7)

Kessler Psychological Distress Scale (K6) - Australian Scoring System

Kessler Psychological Distress Scale (K6) - US Scoring System

Kessler Psychological Distress Scale (K10) - Australian Scoring System

Kessler Psychological Distress Scale (K10) - US Scoring System

Overall Anxiety Severity and Impairment Scale (OASIS)

Patient Health Questionnaire (PHQ-9)

Screen for Child Anxiety Related Disorders (SCARED)

Social and Occupational Functioning Assessment Scale (SOFAS)

Docs: Ingest, label and share model data

Docs: Share data via online repositories

Identify data to be shared

Share data